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What was the impact of the Achieve 3000 literacy program on student reading 
performance, 2013-2014? 


By B. Robert Reeves 


Achieve 3000 was implemented during the 2013-2014 school year. This program was designed to increase 
student Lexile levels. Specifically, Achieve 3000 was targeted at students in the tenth grade in selected 
schools. There were 5,023 tenth-grade students from 17 participating high schools who utilized the web- 
based program. Of these 5,023 students, data were obtained for 4,340 students. This brief was designed to 
analyze the results from the STAAR EOC English II and the Lexile growth as calculated by pre-and post- 
test that Achieve 3000 provided. Results indicate that the program improved student Lexile scores, 
increased EOC English II scale scores, and improved the likelihood of the student passing the EOC English 
II exam at the Phase-in 1 standard both with and without controlling for student characteristics and school 


effects. 


Background 


Achieve 3000 is a web-based differentiated 
K-12 reading program. The program aims to 
raise reading and writing abilities among 
students by determining the Lexile level of the 
student and targeting reading activities to the 
student’s current capabilities. 

This was the first year of the implementation 
of Achieve 3000. During the 2013-2014 school 
year, an estimated 5,023 tenth-grade students 
participated in the program at 17 high schools. 


Data and Methods 


Pre-and post-test results and the number of 
lessons completed by students participating in 
Achieve 3000 were obtained from the 
department of Curriculum, Instruction, and 
Assesment. These results provided Lexile levels 
based only on informational text for the 
participating students, which gave the teachers a 
snapshot of students’ reading ability and can be 
used as a guideline by teachers for appropriate 
reading assignments for each student. Results 
from the STAAR_ End-of-Course (EOC) 
assessment in English II were also analyzed. 


There were two main points of analysis. The 
first examined the effect of Achieve 3000 on the 
STAAR EOC English II results. There were two 
regression models tested for each of the three 
dependent variables (EOC English II Scale 
Score, passing EOC English II at the Phase-in 1 
standard, and passing EOC English II at the 
Phase-in 2 standard). The first regression model 
did not control for student characteristics or 
campus effects, while the second regression 
model controlled for both of these factors. The 
student characteristics included gender, limited 
English proficiency (LEP) status, at-risk status, 
economic status, ethnicity, and whether or not 
students were enrolled in special education. The 
causal variable of interest in all of the regression 
models was the number of activities completed 
in Achieve 3000. 

The second main point of analysis was how 
effective Achieve 3000 was at increasing student 
reading performance as measured by Lexile 
levels. Achieve 3000 calculates student Lexile 
scores based on pre- and post-program tests 
based only on informational text and how the 
student performs during activities. This measure 
was used to determine the impact of the program 
on student reading performance. Two regression 
models were tested using Lexile scores as the 


dependent variable for the 2,782 students. The 
first analyses did not control for school attended 
or student characteristics, as listed above, while 
the second did. Again, the causal variable of 
interest was the number of activities completed 
in Achieve 3000. 

The majority of students at schools that 
utilized the program were historically low 
performing. As illustrated by Figure 1, students 
who attended schools that did not receive 
Achieve 3000 had a 62.7% passing rate at the 
Phase-in 1 standard on the 2013 EOC English I 
Reading, while the schools that did receive the 
program had a 51.3% passing rate. This 
introduces a selection bias and limits the analysis 
of direct HISD comparisons that a random 
assignment would have provided. 

Another limitation relates to schools not 
systematically implementing the program across 
the district. Therefore, all results discussing the 
overall effectiveness of Achieve 3000 are based 
on the program being implemented at the campus 
and not on a specific method of implementation. 


What was the implementation process for 
Achieve 3000? 


This was Achieve 3000’s first year of 
implementation in HISD. Participation was 
voluntary and schools requested and received 
professional development and licenses for the 
Achieve 3000 program. Seventeen campuses 
received licenses: Furr, Jones, Kashmere, Lee, 
Eastwood Academy, Madison, Sterling, Waltrip, 
Washington, Wheatley, Worthing, Sharpstown, 
Scarborough, Westside, Sam Houston Math, 
Science, and Technology, Houston Academy of 
International Studies, and North Forest High. At 
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Figure 1. 2013 English I Reading Passing Rate 
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the school level, each campus selected students 
for program participation based on their own 
criteria. 

The professional development for program 
implementation varied by campus. The first set 
of schools (Eastwood, Jones, Kashmere, Waltrip, 
Houston Academy for International Studies, and 
Worthing high schools) received one day of 
professional development services and a 
maximum platform access for 100 students. The 
next set of schools (Furr, Scarborough, Sterling, 
and Wheatley high schools) received two days of 
professional development services and a 
maximum platform access for 250 students. The 
third group (Lee and Sharpstown high schools) 
received two days of professional development 
services and platform access for a maximum of 
375 students. The next group (Sam Houston and 
Madison high schools) received platform access 
for up to 500 students and three days of 
professional development services. Finally, 
Westside High School received three days of 
professional development training and platform 
access for up to 675 students. Although the 
original contract had a set number of licenses, 
Achieve 3000 allowed additional students to 
receive access at no additional cost to HISD. 

HISD spent a total of $242,315 for use of 
the program and training from July 1, 2013 to 
June 30, 2014. Students who received licenses 
could complete as many activities as desired and 
had access to the program both inside and 
outside of school for the academic year. If each 
license received was used and each student was 
to reach the goal of 40 total activities, the district 
will have paid approximately $1.22 per activity. 
If each student completed 80 activities, the 
district would have paid approximately $0.61 per 
activity. 


What were the demographic characteristics of 


HISD students who participated in Achieve 
3000? 


There were 12,697 tenth-grade students in 
HISD during the 2013-14 schools year. Of these 
students, 5,023 participated in the Achieve 3000 
program. Data were available for 4,340 of the 
participants. There were 2,184 students who 
completed at least one activity in the Achieve 
3000 program, 700 who completed more than 
five activities, 230 completed at least 10 
activities, 59 completed at least 20 activities, and 
only eight completed at least the recommended 
40 activities. 
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Table 1. Demographic Data for Achieve 3000 Students Compared to HISD 10" Grade Students, 2013-2014 


HISD 10th Achieve 3000 10th Completed at least One 
Variables Graders Graders activity 
N N % N % 
Total Enrollment 12,697 4,340 - 2,184 - 
Gender 
Male 6,396 2,304 1,179 54 
Female 6,301 2,036 1,005 46 
Ethnicity 
African American 3,356 A 1,387 32 673 30.8 
Asian/Pacific Islander 469 Shy Vi 1.8 32 1.5 
American Indian 45 0.4 16 0.4 10 0.5 
Hispanic 7,338 57.8 2,605 60 1,391 63.7 
White 1,377 10.8 228 5.3 68 3.1 
Two or More 112 0.9 Dif 0.6 10 0.5 
Econ Dis 9,167 72.2 3,470 80 1,824 83.5 
At-Risk 7,814 61.5 3,026 69.7 1,533 70.2 
Special Education 1,185 9.3 443 10.2 173 7.9 
LEP 1,253 O19 523 12.1 258 11.8 
Gifted/Talented 1,966 15.5 498 11.5 223 10.2 


Source: PEIMS 2013 Fall Snapshot 


Comparing the population of participating 
students to all HISD students, the demographic 
characteristics were similar but not identical (see 
Table 1). More males (53.1%) were represented 
in the program compared to males in the HISD 
student population (50.4%). There was also a 
higher percentage of African American (32% vs. 
26.4%) and Hispanic students (60% vs. 57.8%), 
and a lower percentage of Asian/Pacific Islander 
(1.8% vs. 3.7%) and White students (5.3% vs. 
10.8%) who participated in the program than in 
HISD. Economically disadvantaged, at-risk, 
special education, and LEP students had a higher 
representation in the Achieve 3000 program than 
in HISD, while the gifted and talented students 
had a lower representation. 

There were also no large demographic 
differences between those enrolled in the 
Achieve 3000 program and those who completed 
at least one activity. White, African American, 
special education, LEP, and gifted and talented 
students had a lower rate of activity completion, 
while Hispanic and economically-disadvantaged 
students had a noticeably higher participation 
rate. The specific demographic breakdown is 
presented in Table 1. 


Has usage of Achieve 3000 increased student 
performance on the STAAR EOC English II? 


To assess the ability of Achieve 3000 to 
increase student performance on the STAAR 
EOC English II, the effect on increasing scale 
scores and the probability of the student passing 


at the Phase-in 1 and Phase-in 2 standards was 
examined. The results of these analyses are 
presented in Appendix A, Table 1 (see page 7). 


Scale Score 

The average EOC English II scale score for 
all HISD students during the 2013-14 school 
year was 3826. Non-Achieve 3000 average scale 
scores were 3947 and students who completed at 
least one activity of the Achieve 3000 program 
averaged a scale score of 3841. To pass the EOC 
English II exam at the Phase-in 1 standard, a 
student needed to have at least a 3750 scale 
score. They needed at least a scale score of 3900 
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to pass at the Phase-in 2 standard. As illustrated 
in Figure 2, the average scale score for students 
who completed at least 20 activities (3930) and 
the average scale score for non-Achieve 3000 
students (3947) was higher than the Phase-in 2 
standard, while the average scale score for all 
HISD students (3826) was below the Phase-in 2 
standard but above the Phase-in 1 standard. 
Appendix B, Figure 1 provides a more detailed 
graph of this information showing average scale 
scores for Achieve 3000 students based on the 
number of activities completed (see page 8). 

Graphically, there was a positive correlation 
between scale scores and the number of activities 
completed. Statistically, the number of activities 
had a positive and statistically significant effect 
on the EOC English II scale score regardless of 
school or student characteristics. 

When there were no characteristics 
controlled for, the number of activities 
completed was significant at the p < 0.01 level 
and shows approximately a seven point increase 
in the scale score for each activity completed. 
However, the R? was only .0044. Therefore, the 
model explained less than 1% of the variance in 
scale scores. 

When student characteristics and school 
attended were controlled for, the number of 
activities completed was still significant, though 
only at the p < 0.05 level. The magnitude of the 
effect was reduced to approximately a three point 
increase for each activity completed. However, 
the R’ increased to .51. Therefore, the model 
explains over 50% of the variance in EOC scale 
scores. 


Passing EOC English II at the Phase-in 1 
Standard 

To determine the effectiveness of completed 
activities on whether or not students passed the 
EOC English II exam at the Phase-in 1 standard, 
two probit models were used. The first did not 
control for student characteristics and school 
attended, while the second one did. The results 
and marginal effects were both positive and 
Statistically significant for the number of 
activities completed in both models. These 
results are illustrated in Appendix B, Figure 2 
(see page 8). Figure 3 provides a snapshot of 
Appendix B, Figure 2. It illustrates the 2014 
EOC English II passing rate at the Phase-in 1 
standard for all HISD tenth-grade students, 
tenth-grade students who attended campuses 
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Figure 3. EOC English II Phase-in 1 Passing 


without the Achieve 3000 program, and students 
who completed at least 20 activities of the 
Achieve 3000 program. 

As predicted by the previous 2012-13 EOC 
Reading 1 results presented earlier, the campuses 
without the Achieve 3000 program outperformed 
the campuses with the program. This further 
illustrates that analysis should only include 
campuses that obtained access to Achieve 3000 
due to selection bias. 

Without controlling for students or the school 
attended, activities completed was significant at 
the p < 0.01 level and had a marginal effect of 
.008 which was also significant at the p < 0.01 
level. The pseudo-R’ was also very low at .0038, 
meaning the model explained less than 1% of the 
variance in passing the EOC English II exam at 
the Phase-in 1 standard. 

Once the student characteristics and school 
attended are controlled, the coefficient and 
marginal effects for activities completed both 
decrease, but are still significant at the p < 0.05 
level. The pseudo-R’ for this model is .26, with 
the model explaining 26% of the variance. 

Appendix B, Figure 3 (see page 9) illustrates 
the predicted probability of passing the EOC 
English II exam at the Phase-in 1 standard. The 
predicted probability was created by using the 
probit model which controlled for school 
attended and demographic variables, assigned 
each variable it’s mean value, and allowed the 
activities variable to vary. The predicted 
probability shows what the expected passing rate 
of the EOC English II exam at the Phase-in 1 
standard would have been if each tenth-grade 
student at the schools that received licenses had 
completed the designated number of activities. 
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Figure 4. EOC English II Phase-in 2 Passing 


Passing EOC English II at the Phase-in 2 
Standard 

The same methodology was used to examine 
passing rates at the Phase-in 1 standard was used 
to examine the effectiveness of completed 
activities on whether or not students passed the 
EOC English II exam at the Phase-in 2 standard. 
When school attended and student characteristics 
were omitted, the number of activities completed 
no longer had a statistically significant effect. 
The results are presented in Appendix A, Table 1 
(see page 7) and illustrated in Appendix B, 
Figure 4 (see page 9). Figure 4 provides a 
snapshot of Appendix B, Figure 4. It illustrates 
the 2014 EOC English II passing rate at the 
Phase-in 2 standard for all HISD tenth-grade 
students, tenth-grade students who attended 
campuses without the Achieve 3000 program, 
and students who completed at least 20 activities 
of the Achieve 3000 program. 

Without controlling for student characteristics 
or the school attended, the activities completed 
was significant at the p < 0.05 level and had a 
marginal effect of .004, which was also 
significant at the p < 0.05 level. The pseudo-R* 
was also very low at .0011, meaning the model 
explained less than 1% of the variance in passing 
the EOC English II exam at the Phase-in 2 
standard. 

Once the student characteristics and school 
attended were controlled, the coefficient and 
marginal effects for activities both become 
Statistically insignificant (p = .62). Therefore, 
Achieve 3000 had no statistically significant 
effect on students passing the EOC English II 
exam at the Phase-in 2 standard. 
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HISD Comparison 

Since each campus obtained a limited number 
of Achieve 3000 licenses, a comparison could be 
made with those students who did not have 
access. A two-sample t-test showed a statistically 
significant difference between the students who 
had access to the program (p = 0.00), where 
students with access to the program performed 
better. 

Due to the lack of fidelity in determining 
which students obtain access to the program at 
the individual campuses, the t-test cannot be 
stated as an indicator of program success. 


Cost-Benefit Analysis 

Achieve 3000 has shown to have a positive 
effect on EOC English II outcomes, but the cost 
of the program must be considered when 
analyzing the benefit to the district. As shown 
earlier, there was $242,315 spent on the program 
for the 2013-14 school year. If 40 activities were 
completed for each license, then a total of 
198,000 would have been completed and the 
district would have paid $1.22 per activity 

In practice, only 11,107 activities were 
completed by the students that had data 
available. It cost a total of $21.82 per lesson 
completed, for a gain of approximately three 
points in the scale score, and an increase of about 
half a percentage point to the mean conditional 
probability of a single student passing the EOC 
English II exam at the Phase-in 1 standard. 


Was there an impact on the reading 
performance of students who participated in 
Achieve 3000? 


Appendix B, Figure 5 (see page 10) shows 
student Lexile level by the number of activities 
completed. Figure 5 provides a snapshot of 
Appendix B, Figure 5, and shows the Lexile 
level for all students who had access to Achieve 
3000 and received a Lexile score, completed at 
least one activity, and completed at least 20 
activities. AS seen with student performance on 
the EOC English II exam, participation in 
activities increased student reading outcomes. 

Two regression models measuring the 
impact of completing activities in Achieve 3000 
on student Lexile levels are presented in 
Appendix A, Table 2 (see page 7). 
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Both regression test results showed a 
Statistically significant positive effect of 
activities completed on the Lexile level at the p < 
0.01 level. Without controlling for school effects 
and student characteristics, the model showed a 
10.8 point increase in Lexile level for each 
activity completed with an R? of .02. This model 
explained 2% of the variance. 

Once student characteristics and school 
attended were controlled, the effect reflects a 4.6 
gain in Lexile level for each activity completed, 
with an R*of .25. Therefore, the model then 
explained 25% of the variance. 

Using the same calculations as the previous 
section, the district paid $21.82 for each gain of 
4.6 to the Lexile level per student. 
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Conclusions 


The data show that the Achieve 3000 
program had a positive effect on reading 
outcomes for students that participated in this 
program and completed activities. However, 
current implementation fidelity is questionable. 
This limits the program’s potential impact on 
student reading scores. Given that this is the first 
year of implementation, the program performed 
well for those students who utilized it for at least 
five activities. Future analysis is needed with full 
implementation. 

The current cost per activity is relatively 
high compared to the benefit being derived. 
Regular classroom use and adherence to the 
minimum expected activity completion agreed 
upon before license distribution may greatly 
improve the cost-benefit of the program. 
Program implementation with fidelity targeting 
students who will benefit the most from the 
program should greatly improve tenth-grade 
EOC English II passing rates at the Phase-in 1 
standard. 


For additional information contact the HISD 
Department of Research and Accountability at 


713-556-6700 or e-mail Research@Houstonisd.org. 
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Appendix A: Tables 


Table 1 
Effect of Achieve 3000 on EOC English II Results 


Variables Scale Score Passing EOC English II Phase-in 1 | Passing EOC English II Phase-in-2 
(1) (2) (3) MFX (4) MEX (5) MFX (6) MFX 
Activities 6.68***  2.96** | .019*** .008*** .011** .004** | .01**  .004** 002 001 


Completed (1.56) (1.21) |(.004) (.002) (.005) (.001) | (.004) (.002) (.005) —(.001) 


Controlled: 


Smdem No Yes No No Yes Yes No No Yes Yes 
Characteristics 
High School No Yes No No Yes Yes No No Yes Yes 
N 4148 4,148 | 4,148 4,148 4,148 4,148 
R-Squared 0.004 ol .0038 26 0011 28 


Note: Standard errors are in parenthesis. Columns (1) and (2) are OLS regressions with EOC English II Scale Score as the dependent 
variable. Columns (3) and (4) are probit regressions with whether or not the student passed the EOC English IT exam as the dependent 
variable. For a full table of results including student characteristic and high school outcomes, please contact the HISD’s Department of 
Research and Accountability. 

*p<.10 **p<.05 ***p<.01 


m1) (ed 
Effect of Achieve 3000 on Lexile Scores 


Lexile Score 
(1) (2) 


10.82*** 4.63*** 
(1.35) (1.45) 


R-Squared 


Note: Standard errors are in parenthesis. Columns (1) and (2) are OLS regressions 
with EOC English II Scale Score as the dependent variable. Columns (3) and (4) 
are probit regressions with whether or not the student passed the EOC English II 
exam as the dependent variable. For a full table of results including student 
characteristic and high school outcomes, please contact the HISD’s Department of 
Research and Accountability. 

*p<.10 **p<.05 ***p<.01 


Achieve 3000, 2014 


Appendix B: Figures 


Figure 1 
2014 EOC English II Average Scale Score 
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Figure 5 
Lexile Level as Calculated by Achieve 3000 
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